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SELECTIVE INTERCEPTION OF SYSTEM object code will execute instead. The alternative object code 

CALLS is known as a system call wrapper. 

The interception of system calls is useful to extend and 
customize operating system functionality. For example, the 

BACKGROUND 5 interception of system calls can be used to manipulate 

1 Field of Invention operating system access privileges to provide security 

' . „ beyond that which is provided by the operating system. 

The present invention relates generally to intercepting ^ h ^ mtefc ^ Qf m caUs ^ ide access 

system calls made to multitasking operating systems and tQ ^ fik g esses can be prevented from manipu- 

specifically to selective mterception of system calls made by 10 Mng mes ^ ig desirable> for exam pi e) when a user 

specific processes; wishes to remotely execute a program residing on a web 

2. Background of Invention server, but does not want the remote program to be able to 

Multitasking operating systems such as UNIX and read or alter private data on the user's computer. Today, Java 

Microsoft Windows NT® are widely utilized in commercial applets are commonly employed to provide such security, 

computing systems. Among their many commercial uses, is However, many programs which users wish to remotely 

multitasking operating systems are commonly deployed on execute are written in languages other than Java. System call 

Internet and other network server computers. With the interception allows programs written in any language to be 

popularity and success of the Internet, server computer safely executed remotely. 

operating systems are currently of great commercial impor- The interception of system calls in multitasking operating 

tance. 20 systems is known today, although it is an advanced systems 

Although multitasking operating systems include various programming technique. Multitasking operating system call 

internal resources, it is often desirable to customize or interception is not widely employed in commercial 

extend operating system functionality for a particular use on programming, but select expert systems programmers utilize 

a server computer. Such customization allows a computer the technique. Nonetheless, two serious shortcomings limit 

programmer, a network administrator, or a webmaster to 25 the usefulness of system call interception as it is known 

utilize the operating system in a specific manner beyond the today. 

default system capabilities provided by the manufacturer of first, when a system call is intercepted, the system call 

the operating system. One method of extending and expand- wrapper is executed whenever any process executing under 

ing operating system functionality is the interception of the control of the operating system makes the intercepted 

system calls. system call. No mechanism presently exists to allow selec- 

A system call is a subroutine, the object code of which is tive interception of a system call by only certain processes, 

located in an operating system, such that the subroutine can It would be desirable to selectively intercept system calls 

be called by processes executing under the control of the such that only certain processes execute the wrapper, 

operating system. When executed, a system call performs 35 whereas other processes execute the default system call. For 

some system operation, such as the access of a system example, if file system access calls are intercepted as 

hardware or software resource. Examples of operations described above, no processes will be able to access the 

executed by system calls include reading data from a file, standard file system calls. Although it is desirable for 

opening a network communication channel, and allocating remotely executed processes to be so restricted, this may not 

computer memory to a specific process. Application pro- 4Q be the case for many local processes which should be 

grams (processes) executing under the control of the oper- allowed access to the file system without restriction. Thus, 

ating system call a subroutine (make a system call) in order it is desirable to have a method whereby system calls could 

to bring about the performance of these and other system be selectively intercepted such that only select processes 

operations. execute the system call wrapper when a system call is made. 

In order to make a system call, arguments are program- 45 Another shortcoming with current system call intercep- 

matically loaded into specific registers of the central pro- tion technology is difficulty of development. System call 

cessing unit of the computer on which the operating system wrappers are inserted into the operating system, usually by 

is executing. One of these arguments identifies the specific loading a module into an active operating system kernel, 

system call that is being made. This argument is typically in Thus, system call wrappers execute in a part of computer 

the form of a number that is an offset into the operating 50 memory reserved for the operating system (operating system 

system interrupt vector table, which contains pointers to the address space). 

actual executable code of the system calls. The other loaded System call wrappers, like all computer programs, require 

arguments include parameters to be passed to the system extensive testing and debugging during the development 

call. cycle. When a computer program is being developed and 

Once the arguments have been loaded, a software inter- ss tested, it inevitably generates execution errors and performs 

rupt is generated, signaling to the operating system that a illegal instructions many times before it is debugged and 

process is requesting execution of a system call. The oper- complete. Often, this results in the computer program 

ating system reads the registers, and executes the requested becoming "locked up" because it has overwritten some of its 

system call with the specified parameters. The system call own control memory, or because it is executing an infinite 

executes and performs the desired functionality. If the sys- go 1°°P> or me u ^ e - 

tern call generates a return value, it places the generated Normally, multitasking operating system application pro- 

return value (or a pointer thereto) in a pre-designated reg- grams execute in an area of computer memory reserved for 

ister where it can be accessed by the calling process. non-system processes (user address space). Each program 

In order to intercept a system call, a pointer in an interrupt (process) is assigned, by the operating system, a private 

vector table to a system call is replaced with a pointer to 65 block of computer memory in user address space in which 

alternative object code to be executed instead of the system it can execute. This block of memory is known as the 

call. Then, when the system call is made, the alternative process address space of the associated process. Therefore, 
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when a program generates execution errors during 
development, other processes, and the operating system 
itself, are not effected. Even if an individual process locks 
up, other processes continue to execute, and operating 
system resources can be utilized to terminate the locked 
process so that development may continue. 

System call wrappers execute in operating system address 
space, and therefore system call wrapper execution errors 
effect the entire system. When a wrapper generates execu- 
tion errors, it can overwrite operating system resources such 
as the interrupt vector table or memory map. This requires 
that all processes executing under control of the operating 
system be terminated, and that the operating system be 
restarted. This has the potential to be extremely costly, as 
processes can be executing important commercial function- 
ality (i.e. bank wire transfers) at the time the system termi- 
nates. Even where no critical data is lost, system downtime 
is expensive and undesirable. 

What is needed is a method by which system calls are 
selectively intercepted such that the system call wrapper 
only executes when a system call is made by select pro- 
cesses. When the system call is made by a non-select 
process, the default system call is executed. Additionally, a 
method by which system call wrappers execute in process 
address space would be desirable, in order to avoid the 
expense and other hazards associated with executing system 
call wrappers in operating system address space. 

SUMMARY OF INVENTION 

The present invention allows the selective interception of 
systems calls by specific processes. Additionally, the present 
invention allows a system call wrapper to execute in process 
address space of computer memory. 

In one preferred embodiment, an interception module is 
loaded into the operating system. Pointers in the interrupt 
vector table to system calls to be intercepted are replaced 
with pointers to the interception module. Select processes 
that are to intercept system calls are loaded into process 
address space by a modified loader program. 

A loader program is an operating system utility that is 
used to execute computer programs that are stored on static 
media. Typically, a loader program loads an executable 
image from static media into process address space, and then 
initiates execution of the loaded image by transferring 
execution to the first instruction thereof. 

Like a standard loader program, the modified loader of the 
present invention loads executable images from static media 
into process address space. Additionally, the modified loader 
loads an initialization module and a system call wrapper into 
the process address space of the loaded executable image. 
Each select process that is to intercept system calls is loaded 
by the modified loader program, whereas non-selected pro- 
cesses are loaded with a standard loader. 

Rather than executing the loaded image itself, the modi- 
fied loader executes the loaded initialization module. The 
initialization module registers an entry point in the system 
call wrapper with the interception module. The interception 
module maintains an association table of the select processes 
and system call wrapper entry points In this fashion, the 
interception module can determine the entry point in each 
system call wrapper unique to each user process. 

In an alternative embodiment, a modified loader program 
is utilized to load both selected and non-selected processes. 
In that embodiment, a list of selected processes is stored in 
computer memory. The loader utilizes the list to determine 
if a process to be loaded is selected. If so, the modified 
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loader program loads the process, the system call wrapper, 
and the initialization module, as described above. If the 
process is not selected, the loader simply loads the process 
in the manner of the default loader program. 

5 When an intercepted system call is made by a process, the 
operating system uses the pointer in the interrupt vector 
table to execute the interception module (non-intercepted 
system calls execute normally). The interception module 
determines from the association table whether the calling 

30 process is one of the selected processes with a registered 
entry point in a system call wrapper. If so, the interception 
module prepares to call the appropriate system call wrapper. 

Before calling the system call wrapper, the interception 
module first determines whether the system call was made 

15 by the wrapper, so as to avoid unwanted system call wrapper 
recursion. System call wrappers, like other processes, can 
make system calls. When a wrapper makes a system call, the 
interception module proceeds to call the system call, not to 
recursively call the wrapper, 

20 If the system call was not made by the wrapper, the 
interception module stores an address to which execution 
control is to be returned once execution of the system call 
wrapper has completed. Then, the interception module pro- 
ceeds to call the system call wrapper, which executes in the 

25 process address space of the calling process. 

If it is determined that the process that called the system 
call is not one of the processes which have been selected to 
intercept system calls, the interception module transfers 

3Q execution to the default system call, which executes as if it 
had been called by the process directly. 

Thus, the present invention provides both selective inter- 
ception of system calls by specific processes and execution 
of system call wrappers in process address space. Only 

35 select processes are executed by the modified loader, and 
thus only these select processes have a system call wrapper 
loaded into their process address space. When a system call 
is made by one these select processes, the interception 
module calls the system call wrapper with which the process 

40 is associated. When system calls are made by non-select 
processes, the interception module calls the default system 
call. Therefore, a system administrator of a multitasking 
operating system can determine which select processes will 
intercept which select system calls, thereby overcoming the 

45 above described limitation of the prior art. 

Furthermore, the present invention overcomes the hazards 
associated with executing system call wrappers in operating 
system address space. Because the system call wrappers 
execute in process address space, system call wrapper 

50 execution errors do not effect other processes or the oper- 
ating system itself. This results in an ease of development 
and a level of operating system stability heretofore unavail- 
able using system call wrappers. 

In another embodiment of the present invention, system 

55 call wrappers are loaded into user address space, but not into 
the process address space of any specific process. In this 
embodiment, when a process is loaded into memory the 
modified loader program loads a system call wrapper into 
user address space and executes the loaded wrapper. The 

60 wrapper registers an entry point in itself with the intercep- 
tion module. The interception module updates the associa- 
tion table to include an association between the process 
being loaded and the registered system call wrapper entry 
point. The modified loader then loads and executes the 

65 process. When the process makes a system call, the operat- 
ing system uses the pointer in the interrupt vector table to 
execute the interception module. The interception module 
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determines from the association table whether the calling the operating system kernel 109, while the kernel 109 is 

process is one of the selected processes with a registered active. The interception module 111 is preferably in the form 

entry point in a system call wrapper. If not, the calling of object code, the functional features of which are described 

process is not one of the processes which have been selected in detail below. 

to intercept system calls, so the interception module makes 5 Pointers 114 to system calls 115 are located in an oper- 
the default system call for the process. If so, the interception at ing system interrupt vector table 113. It is to be understood 
module calls the wrapper. that the term "interrupt vector table" as used herein denotes 
This embodiment of the present invention also overcomes an area in operating system address space 105 in which there 
the failings of the prior art. Only select processes are loaded are stored the addresses of system calls. In the UNIX 
by the modified loaded program, and thus only select 10 operating system, this part of the operating system is called 
processes are associated with a system call wrapper. This the "interrupt vector table," and that term is used in this 
facilitates selective interception of system calls by specific specification. Other operating systems employ different ter- 
processes only. Furthermore, the system call wrappers minology to denote the same system component. An inter- 
execute in user address space, thereby overcoming the nipt vector table by any other name is still within the scope 
hazards associated with operating system address space 15 of the present invention. 

execution of system call wrappers. The present invention makes a copy 116 of a pointer 114 
In another embodiment, a system call wrapper executes in to each system call 115 to be intercepted. These copies 116 
operating system address space, but system call interception of pointers 114 are preferably stored in operating system 
is still selective. Interrupt vector table pointers to system -address space 105, but in an alternative embodiments are 
calls to be intercepted are replaced with pointers to the 20 stored in user address space 103. Once the copies 116 have 
system call wrapper, and the original pointers to the system been made and saved, the present invention replaces the 
calls are saved. The system call wrapper maintains an pointers 114 in the interrupt vector table 113 to the system 
identifier table that indicates which processes have been calls 115 to be intercepted with pointers 118 to the inter- 
selected to intercept which system calls. When a system call ception module 111 , such that when a system call 115 to be 
is made, the wrapper executes and determines whether the 25 intercepted is made, the interception module 111 executes 
calling process has been selected to intercept the system call instead. 

that was made. If so, the wrapper executes. Otherwise, the Executing alternative code when a system call 115 is 

wrapper utilizes the saved pointer to make the default made comprises intercepting the system call 115. The steps 

system call. This embodiment also provides for system call ^ 0 f inserting an interception module 111 into the operating 

interception selectivity. system 117, making a copy 116 of an operating system 

pointer 114 to a system call 115, and replacing the operating 

BRIEF DESCRIPTION OF THE DRAWINGS system pointer 114 with a pointer 118 to the interception 

RG. 1 is a block diagram illustrating a system for m <? dule U * facili ^ interc fP tioQ °* * m , ca11 ^ 

selectively intercepting system calls according to a preferred 35 a call xs made to a system call 115 o be intercepted 

embodiment of the present invention. the 0P«tn« f/ff m " 7 ™ cs the pomter 118 m the interrupt 

F . c vector table 113 to the interception module 111 to execute 

RG. 2 is a block diagram illustrating a system for the interception module 111. 

selectively intercepting system calls according to another bc understood ^ m ^ mvcnti Mt dl 

embodiment of the present invention. system ^ n$ ^ be imerceptedt 0my poimers m to 

FIG. 3 is a block diagram illustrating a system for system calls 115 to be intercepted are replaced with pointers 

selectively intercepting system calls according to an alter- n8 {Q me interception modu i e m . Pointers 114 to system 

native embodiment of the present invention. calJs U5 which are not to be intercepted are not replaced. 

HPTATT FH DPSPRTPTTON OF THE lhuS ' When * A 011 "^^?^ svstem 115 * made > the 

d SSSS?^d£iS^ e « s *r cal1 ^TT* "? f;r c T ption , m T n » 

Processes 107 that are selected to intercept system calls 

SYSTEM CALL WRAPPERS IN PROCESS are loaded into process address space 119 by a modified 

ADDRESS SPACE OF SELECTED PROCESSES loader program 121. As explained above, a loader program 

is an operating system utility that is used to execute com- 

FIG. 1 is a high level block diagram illustrating a system 5Q pu t e r programs that are stored on static media. A loader 

for selectively intercepting system calls 115 according to program typically executes in user address space 103. When 

one embodiment of the present invention. In the embodi- a user attempts to execute a computer program (for example 

ment of FIG. 1, system call wrappers 125 execute in the by typing the name of an executable file at a command line, 

process address space 119 of selected processes 107. 0 r by clicking on an icon associated with the program), the 

A computer memory 101 includes user address space 103 55 loader program executes and proceeds to load an executable 

and operating system address space 105, In the user address image from static media into process address space 119, and 

space 103 a process 107 executes. Although FIG. 1 illus- then to initiate execution of the loaded image by transferring 

trates only a single process 107 executing in user address execution to the first instruction thereof, 

space 103, it is to be understood that within a given The present invention utilizes a modified loader program 

computer memory 101, multiple processes 107 can execute 60 121 to load select processes 107 that are to intercept system 

simultaneously, and may have their own system call wrap- calls 115. Like a standard loader program, the modified 

pers 125 and initialization modules 123. loader 121 loads executable images from static media into 

An operating system kernel 109 executes in operating process address space 119. Additionally, the modified loader 

system address space 105. Techniques known in the art are 121 loads an initialization module 123 and a system call 

utilized to insert an interception module 111 into the oper- 65 wrapper 125 into the process address space 119 of the loaded 

ating system 117. In a preferred embodiment, the present executable image. Both the initialization module 123 and the 

invention dynamically loads an interception module 111 into system call wrapper 125 comprise executable object code. 
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The initialization module 121 includes object code to asso- 
ciate the system call wrapper 125 with a select process 107. 
The system call wrapper 125 includes object code to be 
executed instead of intercepted system calls 115. 

Rather than executing the process 107 itself, the modified 5 
loader 121 executes the loaded initialization module 123. 
The initialization module 123 registers an entry point in the 
system call wrapper 125 for the selected process 107 with 
the interception module 111. Preferably, the interception 
module 111 maintains an association table 127 of process 1Q 
identifiers (or in alternative embodiments, alternative pro- 
cess identifying data such as process names) of select 
processes 107 and system call wrapper 125 entry points (or 
in alternative embodiments, alternative system call wrapper 
identifying data such as system call wrapper numbers). The 
initialization module 123 updates the association table 127 
in the interception module 111 by adding an entry compris- 
ing the process identifier of the selected process 107 and the 
entry point in the system call wrapper 125 that is loaded in 
the process address space 119 of the select process 107. 
Thus, when the interception module 111 executes in 
response to the calling of a system call 115, the interception 
module 111 can determine if the calling process 107 is 
associated with a system call wrapper 125. In alternative 
embodiments of the present invention, the associations are ^ 
stored in a format other than a table, for example a linked 
list. In alternative embodiments, the associations are stored 
not in the interception module 1U, but in user address space 
103 or operating system address space 105 as desired. 

The initialization module 123 also creates a small 30 
memory area in the system call wrapper 125 (or alternatively 
in user address space 103 or operating system address space 
105 as desired). This memory area is called the return 
address area 129, and its function is discussed in detail 
below. Additionally, the initialization module 123 sets the 35 
value of an execution flag 131 to indicate that the system call 
wrapper 125 is not currently executing. The execution flag 
131 is preferably located in the system call wrapper 125, but 
may be located in the interception module 111, user address 
space 103, or operating system address space 105 as desired. 4Q 
The initialization module 123 then executes the process 107. 

As stated above, only selected processes 107 are loaded 
by the modified loader program 121. Non-selected processes 
are loaded with the standard, default operating system loader 
program, which simply loads and executes the process. 45 
Thus, non-selected process do not have system calls wrap- 
pers 105 associated therewith. Therefore, selected processes 
107 intercept system calls 115, and non-selected processes 
do not. 

The loading of selected processes 107 and non-selected 50 
processes with two different loader programs is possible 
because multitasking operating systems such as UNIX allow 
the use of multiple loader programs. The decision as to 
which processes 107 are to be loaded with the modified 
loader program 107 can be made by a system administrator, 55 
or by a user. A system administrator can limit access to the 
modified loader program 121, and thus limit the ability of 
users to specify which processes will be selected. 

If a process spawns a child process, the child process is, 
by default, automatically loaded by the loader of the parent 60 
process. Thus child processes of selected processes will, by 
default, be selected processes, and vice versa. Of course, the 
parent process can overwrite the default loader settings, so 
both selected and non-selected processes can spawn both 
selected and non-selected processes as desired. 65 

In an alternative embodiment of the present invention, a 
single, modified loader program 121 is utilized to load both 
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selected processes 107 and non-selected processes. In that 
embodiment, a list of selected processes is stored in com- 
puter memory 101. The list is preferably dynamic, and can 
be updated with additions or deletions as desired. The 
modified loader program 121 utilizes the list to determine if 
a process to be loaded is a selected process 107. If so, the 
modified loader program 121 loads the selected process 107, 
the system call wrapper 125, and the initialization module 
123, as described above. If the process is not selected, the 
modified loader 121 simply loads the process in the manner 
of a default loader program. 

Both selected processes 107 and non-selected processes 
execute in user address space 103 under control of the 
operating system 117. Executing processes make system 
calls 115, When a process makes a system call 115 that is to 
be intercepted, the interception module 111 executes. The 
interception module 111 examines the association table 127 
to determine whether the process that made the system call 
115 is associated with a system call wrapper 125. If the 
process is not so associated, the process is not a selected 
process 107. In this case, the interception module 111 
utilizes the saved copy of the pointer 116 to make the system 
call 115 for the process. If, on the other hand, the process is 
associated with a system call wrapper 125, the process is a 
selected process 107, and the interception module 111 pre- 
pares to execute the system call wrapper 125 which has been 
loaded into the process address space 119 of the calling 
process 107. 

In one embodiment, the interception module 111 first 
examines the execution flag 131 to determine whether the 
system call wrapper 125 is currently executing. If the system 
call wrapper 125 associated with the selected process 107 is 
currently executing, then the system call 115 was made by 
the wrapper 125. Recall that the system call wrapper 125 
executes in the process address space 119 of the selected 
process 107. Therefore, the system call wrapper 125 is 
actually a part of the process 107, and has the same process 
identifier. If a system call 115 is made by the process 107, 
and the section of the process 107 that is executing is the 
system call wrapper 125, then the system call 115 must have 
been made by the wrapper 125. An exception is the special 
case of single processes with multiple threads of execution, 
discussed in detail below. 

If the system call wrapper 125 has made a system call 115, 
it is desirable to execute the actual system call 115, and not 
recursively execute the wrapper 125. When a system call 
wrapper 125 is programmed to make a system call 115, it is 
the intent of the programmer that the actual system call 115 
execute. It is commonly desirable for the system call wrap- 
per 125 to utilize operating system resources. Like any 
process, the system call wrapper 125 utilizes such resources 
by making a system call 115. Thus, when the system call 115 
was made by the wrapper 125, the interception module 111 
uses the saved copy of the pointer 116 to make the system 
call 115. 

On the other hand, if the system call wrapper 125 is not 
currently executing, the interception module 111 must 
execute it. To do so, the interception module 111 first writes, 
to the return address area 129 of the system call wrapper 
125, the address to which to return execution after the 
system call wrapper 125 terminates. This is the address of 
the instruction in the calling process 107 immediately after 
the instruction to make the system call 115. The interception 
module 111 has access to this address because it was pushed 
onto the stack prior to executing the code pointed to by the 
pointer in the interrupt vector table 113. The address is 
pushed onto the stack so that the code can return execution 
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to the address after terminating. In the case of a non- 
intercepted system call 115, the code pointed to is the system 
call 115 itself, which utilizes the pushed address to return 
control to the calling process after executing. However, 
when the system call 115 has been intercepted by the present 5 
invention, the interception module 111 is pointed to rather 
than the system call 115. Because the interception module 
111 will transfer execution to a system call wrapper 125 in 
the process address space 117 of the calling process 107, the 
interception module 111 writes the pushed address to the 10 
return address area 131 of the wrapper 125, so that the 
wrapper 125 can return execution to the proper instruction in 
the calling process 107 once it has terminated. 

Next, the interception module 111 sets the execution flag 
131 to indicate that the system call wrapper 125 is currently 15 
executing. Alternatively the system call wrapper 125 sets the 
flag 131 to so indicate. In alternative embodiments, indica- 
tion that the system call wrapper 125 is currently executing 
is indicated not by a flag but by an alternative indicator, such 
as an entry in a table. Regardless, the interception module 20 
111 transfers execution to the entry point in the system call 
wrapper 125. The system call wrapper 125 proceeds to 
execute in the process address space 117 of the selected 
process 107 that made the system call 115. 

When the system call wrapper 125 finishes executing, it 25 
sets the execution flag 131 (or alternative indicator) to 
indicate that it is not currently executing. Then, the system 
call wrapper 125 transfers execution to the address stored in 
the return address area 129. 

In multitasking operating systems, a single process can 
simultaneously execute multiple threads of execution. 
Therefore, the present invention takes into account the case 
in which a thread of a selected process 107 makes a system 
call 115 to be intercepted, while the system call wrapper 125 35 
associated with the process 107 is executing in response to 
an intercepted system call 115 made by another thread of the 
same process 107. Because the system call wrapper 125 is 
executing, the execution flag 131 will so indicate and thus, 
without more, system calls 115 made by the process 107 will 4Q 
not be intercepted. Yet, it is desirable to intercept system 
calls 115 made a process is 107 thread which is not currently 
executing the system call wrapper 125. 

Threads can be implemented at a process level (user level 
threads) or at an operating system level (kernel level 45 
threads). Kernel level threads utilize operating system 
resources to provide multiple, bona fide threads of execution 
per process. Some operating systems, such as 32 bit versions 
of Microsoft Windows ®, automatically create an indepen- 
dent copy of the global variables of a threaded process for 50 
each thread thereof. Thus, each thread of a selected pro- 
cesses 107 executing under such an operating system has its 
own copy of all of the global variables of the process 107, 
including the execution flag 131. Therefore, if one thread 
intercepts a system call 115 and executes the system call 55 
wrapper 125, the copy of the execution flag 131 pertaining 
to that thread will indicate that the system call wrapper 125 
is executing, and subsequent system calls 115 made by that 
thread will not be intercepted. The copies of the execution 
flag 131 of threads not executing the system call wrapper 60 
125 will indicate accordingly, and thus system calls 115 
made by threads not executing the system call wrapper 125 
will be intercepted. 

Some multitasking operating systems, such as Linux, do 
not automatically provide an independent copy of global 65 
variables for each thread of a process. Embodiments of the 
present invention that execute under the control of such 
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operating systems provide that functionality. Such embodi- 
ments intercept all system calls 115 that create threads, make 
a copy of the global variables of the process for the thread 
being created, and insert a pointer to the created copy in the 
local descriptor table of the thread, the local descriptor table 
being automatically provide by the operating system. 

User level threads do not utilize operating system 
resources, or provide bona fide threads of execution. Instead, 
user level threads simulate multiple threads of execution 
with a set of localized library functions. Because operating 
system resources are not employed to create the threads, the 
operating system has no record of the existence of the 
threads. Therefore, the simulated threads are unable to 
interact with the operating system as if they were actual 
independent threads of execution. Because of this, only one 
system call 115 per process can be serviced at a time, and no 
execution flag 131 conflicts can occur. 

SYSTEM CALL WRAPPERS IN USER 
ADDRESS SPACE 

FIG. 2 is a block diagram illustrating a system for 
selectively intercepting system calls 115 according to 
another embodiment of the present invention. In the embodi- 
ment of FIG. 2, system call wrappers 125 execute in user 
address space 103, but not in the process address space 119 
of any specific process. 

As in the embodiment of FIG. 1, at least one process 107 
executes in user address space 103, an operating system 
kernel 109 executes in operating system address space 105, 
and an interception module 111 is inserted into the operating 
system 117. The present invention makes and saves copies 
116 of pointers 114 to each system call 115 to be intercepted, 
and then replaces the copied pointers 114 with pointers 118 
to the interception module 111. 

As with the embodiment of FIG. 1, selected processes 107 
are loaded into process address space 119 by a modified 
loader program 121. However, the modified loader program 
121 of the embodiment of FIG. 2 operates somewhat dif- 
ferently. The modified loader 121 loads an executable image 
from static media into process address space 119, but does 
not load an initialization module 123. The modified loader 
201 does load a system call wrapper 125 associated with the 
selected process 107, but into user address space 103 instead 
of process address space 119. Thus, each selected processes 
107 has an associated system call wrapper 125, but the 
associated wrapper 125 executes as a separate process. 

Rather than executing the process 107 itself, the modified 
loader 121 executes the system call wrapper 125. The 
wrapper 125 registers an entry point in itself for the selected 
process 107 with the interception module 111. As with the 
embodiment of FIG. 1, the interception module 111 main- 
tains an association table 127 of process identifiers of select 
processes 107 and system call wrapper 125 entry points. The 
initialization module 123 updates the association table 127 
in the interception module 111 by adding an entry compris- 
ing the process identifier of the selected process 107 and the 
entry point in its associated system call wrapper 125. Thus, 
when the interception module 111 executes in response to 
the calling of a system call 115, the interception module 111 
can determine if the calling process 107 is associated with a 
system call wrapper 125. The system call wrapper 125 also 
creates a return address area 129 in itself. Then, the system 
call wrapper 125 returns execution control to the modified 
loader program 121, which executes the selected process 
107. 

As stated above, only selected processes 107 are loaded 
by the modified loader program 121. Non-selected processes 
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are loaded with the standard, default operating system loader embodiment of FIG. 3 loads an executable image and an 

program, which simply loads and executes the process. initialization module 123 into process address space 119, but 

Thus, non-selected process do not have system calls wrap- does not load a system call wrapper 125. The modified 

pers 105 associated therewith. Therefore, selected processes loader 121 proceeds to execute the initialization module 

107 intercept system calls 115, and non-selected processes 5 123, which registers an entry point in the system call 

do not. wrapper 125 for the selected process 107 with the intercep- 

As with the embodiment of FIG. 1, when a process makes tioD module 111. 
a system call 115 the pointer 114 to which has been replaced As with the embodiment of FIG. 1, the interception 
with a pointer 118 to the interception module 111, the module 111 maintains an association table 127 of process 
interception module 111 executes. The interception module 1° identifiers of select processes 107 and system call wrapper 
111 examines the association table 127 to determine whether 125 entry points. The initialization module 123 updates the 
the process that made the system call 115 is associated with association table 127 in the interception module 111 by 
a system call wrapper 125. If the process is not so associated, adding an entry comprising the process identifier of the 
the process is not a selected process, and the interception selected process 107 and the entry point in the associated 
module 111 makes the normal system call 115 for the * 5 system call wrapper 125. Thus, when the interception mod- 
process. If, on the other hand, the process is associated with ule 111 executes in response to the calling of a system call 
a system call wrapper 125, the process is a selected process 115, the interception module 111 can determine if the calling 
107 and the interception module 111 prepares to execute the process 107 is associated with a system call wrapper 125. 
associated system call wrapper 125, using the stored entry Next, the initialization module 123 executes the selected 
point in the system call wrapper 125. 20 process 107. 

In the embodiment of FIG. 2, it is not necessary for the As stated above, only selected processes 107 are loaded 
interception module 111 to examine or set an execution flag by the modified loader program 121. Non-selected processes 
131. Because the system call wrapper 125 executes as a are loaded with the standard, default operating system loader 
separate process, system calls 115 made by the wrapper 125 program, which simply loads and executes the process, 
are not intercepted. The selected process 107 and the system 25 Thus, non-selected process do not have system calls wrap- 
call wrapper 125 have distinct process identifiers, and thus pers 105 associated therewith. Therefore, selected processes 
system calls 115 made by the one can be distinguished from 107 intercept system calls 115, and non-selected processes 
system calls 115 made by the other. Although an association do not. 

is stored in the table 127 between the selected process 107 As with the embodiment of FIG. 1, when a process makes 

and its associated wrapper 125, in this embodiment the 30 an intercepted system call 115, the interception module 111 

selected process 107 and its associated wrapper 125 are two executes. The interception module 111 examines the asso- 

separate processes. The process 107 itself is selected a ciation table 127 to determine whether the process that made 

selected process, but the associated wrapper 125 is not. the system call 115 is associated with a system call wrapper 

Thus, system calls made by the selected process 107 are 125. If the process is not so associated, the process is not a 

intercepted, but those made by the wrapper 125 are not. 35 selected process, and the interception module 111 makes the 

The interception module 111 writes, to the return address system call 115 for the process. If, on the other haod, the 

area 129 of the system call wrapper 125, the address to process is associated with a system call wrapper 125, the 

which to return execution after the system call wrapper 125 process is a selected process 107, and the interception 

terminates. Then, the interception module 111 transfers 40 module 111 executes the associated system call wrapper 

execution to the entry point in the system call wrapper 125. 125. 

The system call wrapper 125 proceeds to execute in user As with the embodiment of FIG. 2, in the embodiment of 

address space 103. When the system call wrapper 125 FIG. 3 it is not necessary for the interception module 111 to 

finishes executing, it transfers execution to the address examine or set an execution flag 131. Because the system 

stored in the return address area 129. 45 call wrapper 125 executes in operating system address space 

105 as opposed to the process address space 119 of a 

SYSTEM CALL WRAPPERS IN OPERATING se]ected ess m> systcm calls m made 5y the wrapper 

SYSTEM ADDRESS SPACE 125 are not intercepted. 

FIG. 3 is a block diagram illustrating a system for In the embodiment of FIG. 3, it is also unnecessary for the 

selective interception of system calls 115 according to 50 interception module 111 to write a return address to a return 

another embodiment of the present invention. In the embodi- address area 129 of a system call wrapper 125. After the 

ment of FIG. 3, system call wrappers 125 execute in system call wrapper 125 terminates, execution is automati- 

operating system address space 105. cally returned to the interception module 111. The system 

As in the embodiment of FIG. 1, at least one process 107 call wrapper 125 and the interception module 111 both 

executes in user address space 103, an operating system 55 execute in operating system address space 105, so no special 

kernel 109 executes in operating system address space 105, steps need be taken to return execution from the system call 

and an interception module 111 is inserted into the operating wrapper 125 to the interception module 111. 

system 117. Additionally, at least one system call wrapper The interception module 111 is called by the selected 

125 is inserted into the operating system 117. Preferably, the process 107, so the interception module 111 has the address 

system call wrapper 125 is loaded into an active kernel 109 60 of the instruction in the calling process 107 to which to- 

as a kernel module. Copies 116 of pointers 114 to each transfer execution after the system call wrapper 125 has 

system call 115 to be intercepted are saved, and then the terminated. Therefore, once the system call wrapper 125 has 

copied pointers 114 are replaced with pointers 118 to the terminated and execution has returned to the interception 

interception module 111. module 111, the interception module 111 returns execution 

As with the embodiment of FIG. 1, selected processes 107 65 control to the selected process 107. 
are loaded into process address space 119 by a modified Thus, to execute the system call wrapper 125, the inter- 
loader program 121. The modified loader program 121 of the ception module 111 simply transfers execution to the entry 
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point in the system call wrapper 125. The system call 
wrapper 125 proceeds to execute in operating system 
address space 105. When the system call wrapper 125 
finishes executing, execution returns to the interception 
module 111, which transfers execution back to the selected 5 
process 107. 

In summary, the present invention allows for selective 
interception of system, by selected processes. In different 
embodiments of the present invention, system call wrappers 
execute in process address space, user address space, and 1Q 
operating system address space as desired. 

What is claimed is: 

1. A method in a computer system for selectively inter- 
cepting system calls made to a multitasking operating, the 
method comprising: 15 

loading an interception module into the operating system; 

replacing pointers to system calls to be intercepted with 
pointers to the interception module, such that calling 
one of the system calls causes the interception module 
to execute; 20 

selecting at least one process to intercept system calls; 

loading a system call wrapper into the process address 
space of at least one selected process; and 

upon execution of the interception module in response to 
a calling of one of the system calls, determining 25 
whether a process that called the system call is a 
selected process. 

2. The method of claim 1 further comprising: 

in response to determining that the process is a selected 
process, executing the system call wrapper loaded in 30 
the process address space of the selected process. 

3. The method of claim 1 further comprising: 

in response to determining that the process is not a 
selected process, executing a default system call. 

4. The method of claim 1 wherein loading the interceptioo 35 
module into the operating system comprises loading the 
interception module into a running operating system kernel. 

5. The method of claim 1 further comprising: 
loading an initialization module into the process address 

space of at least one select process; 40 

executing the initialization module; and 

registering with the interception module, by the initial- 
ization module, an entry point in the loaded system call 
wrapper. 45 

6. The method of claim 5 wherein the loading of at least 
one system call wrapper and the loading of the initialization 
module are performed by a modified operating system 
loader program, and the method further comprises: 

utilizing the modified loader program to load at least one 5Q 
selected process into process address space. 

7. The method of claim 1 further comprising: 

storing an indicator of an association between a loaded 
system call wrapper and a selected process. 

8. The method of claim 7 wherein the storing further 55 
comprises: 

updating an association table in the interception module. 

9. The method of claim 8 wherein: 

determining whether a process that called the system call 
is a selected process comprises determining if an iden- 60 
tifier of the process that called the system call is 
included in the association table. 

10. The method of claim 1 further comprising: 

upon execution of the interception module in response to 
a calling of one of the system calls by a selected 65 
process, storing an address to which to return execu- 
tion; and 



upon completion of execution of the system call wrapper, 
returning execution to the stored address. 

11. The method of claim 10 wherein the address to which 
to return execution is stored in the system call wrapper. 

12. The method of claim 1 further comprising: 

in response to determining that the process is a selected 
process, determining whether the system call wrapper 
loaded in the process address space of the selected 
process is currently executing. 

13. The method of claim 12 further comprising: 

in response to a determination that the system call wrap- 
per is not currently executing, executing the system call 
wrapper. 

14. The method of claim 1 further comprising: 

upon loading the system call wrapper, setting an indicator 
to indicate that the system call wrapper is not currently 
executing. 

15. The method of claim 12 further comprising: 

prior to executing the system call wrapper, setting an 
indicator to indicate that the system call wrapper is 
currently executing. 

16. The method of claim 12 further comprising: 

upon completion of execution of the system call wrapper, 
setting an indicator to indicate that the system call 
wrapper is not currently executing. 

17. The method of claim 12 further comprising: 

in response to a determination that the system call wrap- 
per is currently executing, executing a default system 
call. 

18. A method in a computer system for selectively inter- 
cepting system calls made to a multitasking operating 
system, the method comprising: 

loading an interception module into the operating system; 

replacing pointers to system calls to be intercepted with 
pointers to the interception module, such that calling 
one of the system calls causes the interception module 
to execute; 

selecting at least one process to intercept system calls; 
loading at least one system call wrapper into user address 
space; 

for each loaded system call wrapper, associating the 
system call wrapper with a selected process; and 

upon execution of the interception module in response to 
a calling of one of the system calls, determining 
whether a process that called the system call is a 
selected process. 

19. The method of claim 18 further comprising: 

in response to determining that the process is a selected 
process, executing the associated system call wrapper 
in user address space. 

20. The method of claim 18 further comprising: 

in response to determining that the process is not a 
selected process, executing a default system call. 

21. The method of claim 18 wherein: 

loading the interception module into the operating system 
comprises 

loading the interception module into a running operat- 
ing system kernel. 

22. The method of claim 18 wherein: 

associating the system call wrapper with a select process 
comprises 

registering with the interception module, by the loaded 
system call wrapper, an entry point in the system call 
wrapper. 
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23. The method of claim 18 wherein the loading of at least 
one system call wrapper is performed by a modified oper- 
ating system loader program, and the method further com- 
prises: 

utilizing the modified loader program to load at least one 
selected process into process address space. 

24. The method of claim 18 further comprising: 
storing an indicator of an association between a loaded 

system call wrapper and a selected process. 

25. The method of claim 24 wherein the storing further 
comprises: 

updating an association table in the interception module. 

26. The method of claim 25 wherein: 

determining whether a process that called the system call 
is a selected process comprises determining if an iden- 
tifier of the process that called the system call is 
included in the association table. 

27. The method of claim 18 further comprising: 

upon execution of the interception module in response to 
a calling of one of the system calls by a selected 
process, storing an address to which to return execu- 
tion; and 

upon completion of execution of the system call wrapper, 
returning execution to the stored address. 

28. The method of claim 27 wherein the address to which 
to return execution is stored in the system call wrapper. 

29. A method in a computer system for selectively inter- 
cepting system calls made to a multitasking operating 
system, the method comprising: 

replacing pointers to system calls to be intercepted with 
pointers to object code, such that calling one of the 
system calls causes the object code to execute; 

selecting at least one process to intercept system calls; 

loading at least one system call wrapper into user address 
space; and 

upon execution of the object code in response to a calling 
of one of the system calls, determining whether a 
process that called the system call is a selected process. 

30. The method of claim 29 further comprising: 

in response to determining that the process is a selected 
process, executing a system call wrapper in user 
address space. 

31. The method of claim 29 further comprising: 

in response to determining that the process is not a 
selected process, executing a default system call in 
operating system address space. 

32. The method of claim 29 further comprising: storing an 
indicator of an association between a loaded system call 
wrapper and a selected process. 

33. A method in a computer system for selectively inter- 
cepting system calls made to a multitasking operating 
system, the method comprising: 

loading a system call wrapper into operating system 

address space; 
replacing pointers to system calls to be intercepted with 

pointers to the system call wrapper, such that calling 

one of the system calls causes the system call wrapper 

to execute; 

selecting at least one process to intercept system calls; 
determining whether a process that called a system call is 

a selected process; and 
in response to a determination that the process that called 

the system call is a selected process, executing the 

system call wrapper. 
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34. The method of claim 33 wherein loading a system call 
wrapper into operating system address space comprises 
loading the system call wrapper into a running operating 
system kernel. 

35. The method of claim 33 further comprising: 
storing a list of identifiers of selected processes for which 

system calls will be intercepted; and 
wherein determining whether a process that called a 
system call is a select process comprises determining if 
an identifier of the process that called the system call is 
included in the list. 

36. The method of claim 33 further comprising: 
saving pointers to system calls to be intercepted; and 

in response to a determination that the process that called 
the system call is not a select process, utilizing a saved 
pointer to execute the system call. 

37. A computer program product for selectively intercept- 
ing system calls made to a multitasking operating, the 
computer program product comprising: 

program code for loading an interception module into the 
operating system; 

program code for replacing pointers to system calls to be 
intercepted with pointers to the interception module, 
such that calling one of the system calls causes the 
interception module to execute; 

program code for selecting at least one process to inter- 
cept system calls; 

program code for loading a system call wrapper into the 
process address space of at least one selected process; 

program code for, upon execution of the interception 
module in response to a calling of one of the system 
calls, determining whether a process that called the 
system call is a selected process; and 

a computer readable medium on which the program codes 
are stored. 

38. The computer program product of claim 37 further 
comprising: 

program code for, in response to determining that the 
process is a selected process, executing the system call 
wrapper loaded in the process address space of the 
selected process. 

39. The computer program product of claim 37 further 
comprising: 

program code for, in response to determining that the 
process is not a selected process, executing a default 
system call. 

40. The computer program product of claim 37 further 
comprising: 

program code for loading the interception module into a 
running operating system kernel. 

41. The computer program product of claim 37 further 
comprising: 

program code for loading an initialization module into the 

process address space of at least one select process; 
program code for executing the initialization module; and 
program code for registering with the interception 
module, by the initialization module, an entry point in 
the loaded system call wrapper. 

42. The computer program product of claim 41 further 
comprising: 

program code comprising a modified loader program for 
the loading of at least one system call wrapper and the 
loading of the initialization module. 

43. The computer program product of claim 37 further 
comprising: 
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program code for storing an indicator of an association program code for selecting at least one process to inter- 
between a loaded system call wrapper and a selected cept system calls; 

process. program code for loading at least one system call wrapper 

44. The computer program product of claim 43 further into user address space; 

comprising: 5 program code for associating each loaded system call 

program code for updating an association table in the wrapper with a selected process; 

interception module. program code for, upon execution of the interception 

45. The computer program product of claim 44 further module in response to a calling of one of the system 
comprising: calls, determining whether a process that called the 

program code for determining whether a process that io system call is a selected process; and 

called the system call is a selected process comprises a computer readable medium on which the program codes 

determining if an identifier of the process that called the are stored. 

system call is included in the association table. 55. The computer program product of claim 54 further 

46. The computer program product of claim 37 further comprising: 

comprising: 15 program code for, in response to determining that the 

program code for, upon execution of the interception process is a selected process, executing the associated 

module in response to a calling of one of the system system call wrapper in user address space. 

calls by a selected process, storing an address to which 56. The computer program product of claim 54 further 

to return execution; and comprising: 

program code for, upon completion of execution of the 20 program code for, in response to determining that the 

system call wrapper, returning execution to the stored process is not a selected process, executing a default 

address. system call. 

47. The computer program product of claim 46 further 57. The computer program product of claim 54 further 
comprising: comprising: 

program code for storing, in the system call wrapper, the 25 program code for loading the interception module into the 

address to which to return execution. operating system comprises loading the interception 

48. The computer program product of claim 37 further module into a running operating system kernel, 
comprising: 58. The computer program product of claim 54 further 

program code for, in response to determining that the comprising: 

process is a selected process, determining whether the 30 program code for associating the system call wrapper with 

system call wrapper loaded in the process address space a select process comprises registering with the inter- 

of the selected process is currently executing. ception module, by the loaded system call wrapper, an 

49. The computer program product of claim 48 further entry point in the system call wrapper, 
comprising: 59, Hie computer program product of claim 54 further 

program code for, in response to a determination that the 35 comprising: 

system call wrapper is not currently executing, execut- program code for loading of at least one system call 

ing the system call wrapper. wrapper by a modified operating system loader pro- 

50. The computer program product of claim 37 further gram. 

comprising: 60. The computer program product of claim 54 further 

program code for, upon loading the system call wrapper, comprising: 

setting an indicator to indicate that the system call program code for storing an indicator of an association 

wrapper is not currently executing. between a loaded system call wrapper and a selected 

51. The computer program product of claim 48 further process. 

comprising: 45 61. The computer program product of claim 60 further 

program code for, prior to executing the system call comprising: 

wrapper, setting an indicator to indicate that the system program code for updating an association table in the 

call wrapper is currently executing. interception module. 

52. The computer program product of claim 48 further 62. The computer program product of claim 61 further 
comprising: 50 comprising: 

program code for, upon completion of execution of the program code for determining whether a process that 

system call wrapper, setting an indicator to indicate that called the system call is a selected process comprises 

the system call wrapper is not currently executing, determining if an identifier of the process that called the 

53. The computer program product of claim 48 further system call is included in the association table, 
comprising: 5S 63. The computer program product of claim 54 further 

program code for, in response to a determination that the comprising: 

system call wrapper is currently executing, executing a program code for upon execution of the interception 

default system call. module in response to a calling of one of the system 

54. A computer program product for selectively intercept- calls by a selected process, storing an address to which 
ing system calls made to a multitasking operating system, 60 10 retum execution; and 

the computer program product comprising: program code for, upon completion of execution of the 

program code for loading an interception module into the system call wrapper, returning execution to the stored 

operating system; address, 

program code for replacing pointers to system calls to be 64. The computer program product of claim 63 further 

intercepted with pointers to the interception module, 65 comprising: 

such that calling one of the system calls causes the program code for storing, in the system call wrapper, the 

interception module to execute; address to which to return execution. 
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65. A computer program product for selectively intercept- 
ing system calls made to a multitasking operating system, 
the computer program product comprising: 

program code for replacing pointers to system calls to be 
intercepted with pointers to object code, such that 5 
calling one of the system calls causes the object code to 
execute; 

program code for selecting at least one process to inter- 
cept system calls; 

program code for loading at least one system call wrapper 
into user address space; 

program code for upon execution of the object code in 
response to a calling of one of the system calls, 
determining whether a process that called the system i$ 
call is a selected process; and 

a computer readable medium on which the program codes 
are stored. 

66. The computer program product of claim 65 further 
comprising: 20 

program code for, in response to determining that the 
process is a selected process, executing a system call 
wrapper in user address space. 

67. The computer program product of claim 65 further 
comprising: 25 

program code for, in response to determining that the 
process is not a selected process, executing a default 
system call in operating system address space. 

68. The computer program product of claim 65 further 
comprising: 30 

program code for storing an indicator of an association 
between a loaded system call wrapper and a selected 
process. 

69. A computer program product for selectively intercept- 35 
ing system calls made to a multitasking operating system, 
the computer program product comprising: 
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program code for loading a system call wrapper into 
operating system address space; 

program code for replacing pointers to system calls to be 
intercepted with pointers to the system call wrapper, 
such that caOing one of the system calls causes the 
system call wrapper to execute; 

program code for selecting at least one process to inter- 
cept system calls; 

program code for determining whether a process that 
called a system call is a selected process; 

program code for, in response to a determination that the 
process that called the system call is a selected process, 
executing the system call wrapper; and 

a computer readable medium on which the program codes 
are stored. 

70. The computer program product of claim 69 further 
comprising: 

program code for loading the system call wrapper into a 
running operating system kernel. 

71. Tbc computer program product of claim 69 further 
comprising: 

program code for storing a list of identifiers of selected 
processes for which system calls will be intercepted; 
and 

program code for determining if an identifier of the 
process that called the system call is included in the fist. 

72. The computer program product of claim 69 further 
comprising: 

program code for saving pointers to system calls to be 

intercepted; and 
program code for, in response to a determination that the 

process that called the system call is not a select 

process, utilizing a saved pointer to execute the system 

call. 

* * * * * 
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